Mastering Apache Solr 7.x: An expert guide to advancing, optimizing, and scaling your enterprise search by Sandeep Nair & Chintan Mehta & Dharmesh Vasoya

Author:Sandeep Nair & Chintan Mehta & Dharmesh Vasoya [Nair, Sandeep]
Language: eng
Format: epub
Tags: COM018000 - COMPUTERS / Data Processing, COM005030 - COMPUTERS / Enterprise Applications / Business Intelligence Tools, COM021000 - COMPUTERS / Databases / General
Publisher: Packt Publishing
Published: 2018-02-21T23:00:00+00:00

Language identification

At the time of indexing, language identification is required to map text to language-specific fields. Solr uses the langid UpdateRequestProcessor for language identification. Two types of UpdateRequestProcessor are provided by Solr.

TikaLanguageIdentifierUpdateProcessor: Uses the language identification libraries in Apache Tika

LangDetectLanguageIdentifierUpdateProcessor: Uses the open source Language Detection Library for Java

Configuration of TikaLanguageIdentifierUpdateProcessor in solrconfig.xml:

<processor class="org.apache.solr.update.processor.TikaLanguageIdentifierUpdateProcessorFactory">

<lst name="defaults">

<str name="langid.fl">title,subject,text,keywords</str>

<str name="langid.langField">language_s</str>

</lst>

</processor>

Configuration of LangDetectLanguageIdentifierUpdateProcessor in solrconfig.xml:

<processor class="org.apache.solr.update.processor.LangDetectLanguageIdentifierUpdateProcessorFactory">

<lst name="defaults">

<str name="langid.fl">title,subject,text,keywords</str>

<str name="langid.langField">language_s</str>

</lst>

</processor>

UpdateRequestProcessor provides many langid parameters. We are not going to explain them here. Determining the language at index time is always preferable over query time. During query time, the input provided by the user may be short and sometimes not meaningful enough to extract language information. At index time, full documents are present, so language identification becomes easier.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.